Is this flaky, or is it just dead? I may have hit this on every run since the rebase ;).

unit:

FAIL: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kuberuntime TestCreatePodSandbox_RuntimeClass/missing_RuntimeClass 0s
FAIL: github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kuberuntime TestCreatePodSandbox_RuntimeClass 100ms

although I don't see how that would be due to f56f493, so

/retest

/lgtm

for good measure ;).

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: smarterclayton, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

It succeeds extremely rarely. It just so happened that was the two prior runs to the rebase, so we thought it was gravy.

/test e2e-aws

Is it worth kicking until the batch job resolves?

I'm waiting for a green e2e-aws and then I'm going to force merge

/retest

This requires kublet-1.12 because it is a new feature that depends on volume plugin being registered on the node. So far flakes I have investigated related to this are still using 1.11 kubelet. But I am still looking.

e2e-aws-builds:

Failing tests:

[Feature:Builds][Slow] openshift pipeline build jenkins-client-plugin tests using the ephemeral template [Suite:openshift]

/test e2e-aws-builds

/test e2e-aws

e2e-aws:

2019/01/26 03:40:18 Ran for 1m15s
error: could not run steps: could not wait for template instance to be ready: could not determine if template instance was ready: failed to create objects: object is being deleted: pods "e2e-aws" already exists

/test e2e-aws

/retest

Please review the full test history for this PR and help us cut down flakes.

/retest

I'm waiting for a green e2e-aws and then I'm going to force merge

Looking at origin's e2e-aws history, we haven't had anything pass in 11+ hours. So if you don't mind waiting it out while whatever the current batch job is fails, maybe this will just get merged without having to force it. Or maybe Tide will decide it needs to retest it in zounds of possible batch combinations, I dunno ;).

It's possible this run is dying with:

level=error msg="\t* aws_iam_role.bootstrap: Error creating IAM Role ci-op-2gqyb832-55c01-bootstrap-role: EntityAlreadyExists: Role with name ci-op-2gqyb832-55c01-bootstrap-role already 

although I don't know why that would still be running after an hour. You may want to bump your commit timestamp or something to get a fresh, new namespace.

I’m more concerned that teardown isn’t running. Is this the installer backgrounding?

/test e2e-aws

No teardown logs from this last run, but here's a normal teardown from a recent installer-PR failure:

$ curl -s https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_in
staller/1129/pull-ci-openshift-installer-master-e2e-aws/3174/artifacts/e2e-aws/container-logs/tear
down.log.gz | gunzip | tail -n 3
level=info msg=Deleted arn="arn:aws:ec2:us-east-1:460538899914:security-group/sg-0b6ccc03f79dbbb5f
" id=sg-0b6ccc03f79dbbb5f
level=info msg=Deleted arn="arn:aws:ec2:us-east-1:460538899914:vpc/vpc-03b6fdd1c55f21f19" id=vpc-0
3b6fdd1c55f21f19
level=info msg=Deleted arn="arn:aws:ec2:us-east-1:460538899914:dhcp-options/dopt-0616c23780c3426ec
" id=dopt-0616c23780c3426ec

My guess is the issue was job 2885 or a similar run. setup failed, but then test was killed:

level=fatal msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: failed to apply using Terraform"
2019/01/26 04:08:59 Container setup in pod e2e-aws failed, exit code 1, reason Error
Another process exited
2019/01/26 04:09:14 Container test in pod e2e-aws failed, exit code 1, reason Error
{"component":"entrypoint","level":"error","msg":"Entrypoint received interrupt: terminated","time":"2019-01-26T05:21:56Z"}
2019/01/26 05:21:56 error: Process interrupted with signal interrupt, exiting in 2s ...
2019/01/26 05:21:56 cleanup: Deleting template e2e-aws

My guess is artifact gathering was slow "no cluster", and we reaped teardown before it completed. But I don't seee any teardown logs, so hard to know.

I’ve noticed some of that in jobs today.

/retest

We should short-circuit artifact gathering when Terraform fails. Have the installer exit 2? Grep the logs?

Ah, or gate on some really-basic API call succeeding. I can work that up tomorrow if you don't beat me to it ;).

I’ll probably be looking at other things so don’t worry about me also looking at it :)

What the fork.

One risk is that an api failure flake could result in no logs. We should definitely consider something like queue terminating early if enough sequential things aren’t gathered.

@smarterclayton: The following test failed, say /retest to rerun them all:

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.